Device for intuitive dexterous touch and feel interaction in virtual worlds

ABSTRACT

A device for dexterous interaction in a virtual world in disclosed. The device includes a housing including a plurality of buttons and a plurality of vibration elements each associated with at least one of the plurality of buttons. An orientation sensor detects orientation of the housing, and a bearing is configured to allow the housing to freely rotate in a plurality of directions. A processor is in communication with the plurality of buttons, the plurality of vibration elements, and the orientation sensor. A transmitter/receiver unit is configured to receive data from the processor and configured to send and receive data from a central processing unit.

INCORPORATION BY REFERENCE

The following documents are incorporated herein by reference as if fullyset forth herein: U.S. Provisional Application 62/172,061 filed Jun. 6,2015, and U.S. Provisional Application 62/080,759, filed Nov. 17, 2014.

FIELD OF INVENTION

The present application relates to a controller, and more specificallyrelates to a feedback controller for the virtual reality environment.

BACKGROUND

Humans use their hands, and most particularly their fingers and palms,to physically manipulate and sense objects in and around theirenvironment without exerting much effort. The two primary functions ofhands can be broken down into gross motor skills (such as grasping alarge object) and fine motor skills (such as picking up a small pebble).Hands also have a tactile sensing ability, which allows a human todetect the surface features of an object. One of the newest developmentsin 3-D technology involves the emergence of low cost, high definitionstereo headsets. These headsets present the left and right eyes withstereo images and thereby can create the illusion of being inside acomputer generated 3-D world. Head-mounted displays for immersivesoftware have been available for some time now, but the newer headsetsimprove the quality of the 3-D image presented to the user and lower thecost making it available to most users.

It is expected that the emergence of low cost headsets and similar newimmersive technologies will sharply increase the number of gamers thatwill be entering the marketplace for 3-D environment games andapplications. However, the technology that allows these users tointeract with the 3-D world is lagging behind. The most direct,instinctive and effective way of interacting is through the use ofhands. Currently, one type of interactive gaming technology includesdata gloves that can be used by gamers to interact with objects. Datagloves are equipped with sensors that detect the movements of the handand fingers and interface those movements with a 3-D system running on acomputer. When data gloves are used in a virtual reality environment,the user sees an image of a corresponding hand and the user canmanipulate objects in the virtual environment using the glove.Unfortunately, these existing data gloves are expensive, cumbersome, andoften provide an inaccurate replication of a user's hand movements. Theexisting data gloves require that the user physically wears a glove,sometimes for extended periods of time, which can cause discomfort forthe user while the user raises their hands and arms to gesture and tomanipulate objects. Current data gloves also include complex sensorsystems to collect data related to the positions of the fingers and thelocation and orientation of the hand, which further increases the costsof these devices.

Various systems have been proposed that include cameras to capture imagedata that includes the hands of the users. This image data is convertedinto parametric data for use in software such as games or 3-D virtualworld visualization systems. Systems of the current art however requirethat the hand is always in the field of view of cameras and that camerasare either stationary and in a location independent from the user's bodyor worn on the user's head so that the field of view of the cameras doesnot necessarily include the user's hand.

Neither the data gloves nor the vision-based technologies of the presentart allow systems to give force feedback to the user's hand. Forexample, if the user uses a data glove to reach out and touch a tree invirtual space the data glove system will not physically stop the handwhen it contacts the tree. If users move their virtual hands up and downagainst the tree the current data glove systems do not provide aphysical feedback to the user's hand that conveys the texture orroughness of the tree's bark.

SUMMARY

The present invention is directed, generally, to an inexpensive systemthat allows a user to comfortably and intuitively control articulation,location, and orientation of a virtual hand in a virtual world andprovides physical force feedback to the user's hand related tointeractions between the virtual hand and virtual objects in the virtualworld. The system of the present application comprises a handheldhousing which fits in the palm of a user's hand and comprises aplurality of pressure-sensitive buttons which are positioned to restunder the user's respective fingers as the user grips the handheldhousing.

As the user presses a specific finger or fingers onto associatedbuttons, then that particular finger or fingers associated with thecorresponding virtual hand in a virtual environment is manipulated toreflect the user's movements. For example, if the user presses aspecific finger on a respective physical button, then the virtual hand'sassociated finger will also grip and/or bend. As the user releases thebutton, then the finger of the associated virtual hand in virtual spaceopens and/or unbends. The button may be equipped with a sensorconfigured to detect the degree (e.g., depth or strength of pressing)with which the button is pressed, and the associated virtual finger willbend to reflect the finger's movement against the button as detected bythe sensor.

In one embodiment, the handheld housing of the device of the presentinvention furthermore includes a gyroscope for detecting multipledegrees of motion, with as full a range of motion as possible, and fordetecting orientation with respect to the housing of the handhelddevice. The housing of the handheld device is attached to the grip on a3-D computer mouse and force feedback controller which controller isfixed in space. A bearing attaches the housing to the 3-D computer mouseand force feedback controller allows the housing to freely move in threedimensions and through a range of motion. Therefore, the user can changethe orientation of the virtual hand in the virtual space by intuitivelychanging the orientation of the orientation of the housing of thehandheld device. Since the housing of the handheld device is physicallyattached, through the bearing, to the grip on a 3-D computer mouse andforce feedback controller, the user can move the handset in threedimensions and through a range of motion to change the associatedlocation of a virtual hand in a virtual space. The buttons of thehousing of the handheld device include an actuator and vibrator forimparting onto the fingers of the user tactile feedback related to datagenerated by the 3-D visualization software of the virtual world.

The buttons and the gyroscope are connected to a processor in thehousing of the handheld device. The processor creates, collects,processes, and/or transmits data related to the articulation of thevirtual hand to a processor for use in software that uses such data.That software can be a 3-D visualization system, such as a 3-D worldviewed through a 3-D headset. The software could be implemented in known3-D headsets, such as Oculus Rift and Samsung Gear VR. The software alsogenerates tactile data for actuating and vibrating the buttons inrelation with and/or in response to the user's experience in the virtualworld. The tactile data is transmitted to the processor for controllingsaid buttons. Thus, the system translates the movement of a user's handso that it is reflected in a virtual world, and also translates actionsin the virtual world back to a user's hand through feedback.

The present device is configured to be operated by a human hand forcontrolling the placement, movement, articulation, and gestures for acorresponding virtual hand in a virtual world and provides tactilefeedback to the user from the associated virtual world. According to anaspect of the invention, the articulation of a hand may be consideredthe positioning of palm and the fingers in space through a range ofmotion. The device is preferably compact, such the device fits in thepalm of the hand and when the hand naturally and comfortably grasps thedevice, the fingers of the hand rest on buttons. An ergonomic design ispreferred. The device preferably has at least five primary buttons, onecorresponding to each finger. However, one of ordinary skill in the artwould recognize from this disclosure that any number of buttons could beprovided. For example, one embodiment can include three or four buttonsper finger, to accommodate each segment of a user's respective fingerand the related movement of the phalanges. The buttons are preferablymovement and/or pressure sensitive so that the user can flex individualfingers by slightly pressing and releasing buttons associated with eachfinger. Depending on the degree of pressure applied, the correspondingmotion of the associated finger will vary. The buttons of the devicealso preferably vibrate and move to provide tactile feedback to the userwhich is associated with touch experienced by fingers in the virtualworld associated with each button. In other words, the device allows thevirtual hand to send sensory output to the user's actual hand.

The device of the present invention senses changes in the orientation ofthe handheld device so that the user can naturally and intuitivelycontrol the orientation of the hand in the virtual world. To accommodateany variations in the orientation and manipulation of the device by theuser's hand, the device preferably includes an accelerometer orgyroscope for detecting rotation of the device. In addition, the deviceattaches to a 3-D computer mouse and force feedback controller. A 3-Dmouse and feedback controller is a combined device that has a grip whichcan be held and moved in three dimensions by a user to enter datarelated to 3-D position into a computer. The 3-D mouse and feedbackcontroller include motors that can impart forces onto the grip so thatthe user receives force feedback from software running in the computer.Accordingly, the location of the device of the present invention in 3-Dspace can be detected and transmitted to virtual world software whichallows the user to control the location of a virtual hand and allowsforce feedback to be applied to the user's hand through the device.

The device of the present invention is designed to limit the actualamount of movement of the user's hand and software implementing relativepositioning algorithms serves to move the user's virtual hand in thevirtual world through its entire range of motion. This promotes ease ofuse of the system, particularly limiting use of the user's arm andshoulder.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a shows a side view of the device according to one embodiment.

FIG. 1b shows a bottom view of the device according to one embodiment.

FIG. 1c shows an alternative embodiment of a housing of the device.

FIG. 1d shows a schematic diagram of an embodiment of the housing of thedevice.

FIG. 2 shows an internal view of components within the device of FIGS.1a -1 d.

FIG. 3 shows a perspective view of the device including degrees ofmovement axes.

FIG. 4 shows a diagram of the device of FIGS. 1a, 1b , and 2 with acomputer system according to one embodiment.

FIG. 5 shows a flow chart illustrating the steps for interaction betweenthe device and the computer according to one embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As shown in FIG. 2, a first embodiment of the device 100 includes ahousing 1 which is configured to fit comfortably in the palm of theuser's hand 200. The housing 1 preferably has a generally ellipsoidshape, which corresponds to a user's palm and closed first. Preferably,the housing 1 has an ergonomic shape providing for fit and comfort. Thehousing may comprise or be otherwise covered by a flexible or elasticmaterial, such as a rubberized material, or NEOPRENE, for comfort. Thehousing 1 preferably includes a plurality of buttons 2, 4, 6, 8, 10. Inone embodiment, the housing 1 includes five buttons 2, 4, 6, 8, 10 whichare positioned on the housing 1 such that when the user grasps thehousing 1 with their palm positioned comfortably, the buttons 2, 4, 6,8, 10 rest, in one embodiment, under the tips of the fingers of theuser's hand. As shown in FIG. 1c , in a multiple buttons embodiment, abutton may rest under each section or portion of a user's finger. Asshown in FIG. 1c , the housing 1′ includes a plurality of finger buttons2 a, 2 b, 4 a, 4 b, 6 a, 6 b, 8 a, 8 b, 10 a, 10 b, as well as a buttonfor the palm 23.

Those of skill in the art will recognize from this disclosure that adifferent number and configuration of buttons could be used within thescope of the invention. For example, in one embodiment, the housing 1can include, by way of illustration, fifteen buttons, with a five setsof three buttons, each of the five sets configured for engagement with arespective finger. In another embodiment, a track-ball or scroll-wheelcan be provided on the housing 1 for manipulation by a user's hand. Eachof the plurality of buttons 2, 4, 6, 8, 10 is associated with acorresponding one of a plurality of vibration elements 3, 5, 7, 9, 11.In one embodiment, each of the plurality of buttons 2, 4, 6, 8, 10 isarranged directly in physical engagement with a respective one of theplurality of vibration elements 3, 5, 7, 9, 11.

The plurality of vibration elements 3, 5, 7, 9, 11 causes its associatedbutton 2, 4, 6, 8, 10 to vibrate, which imparts tactile feedback to thefinger associated with said button 2, 4, 6, 8, 10. In one embodiment,the housing 1 furthermore comprises an orientation sensor 30 preferablya gyroscope 30 a in communication with or otherwise comprising a sensor29, for detecting and measuring the orientation of the housing 1. In oneembodiment, the orientation sensor 30 includes a gyroscope 30 a and anaccelerometer 30 b. In another embodiment, the orientation sensor 30 canalso include a heartrate sensor. The gyroscope 30 a is configured todetect a plurality of types of motion, including but not limited todisplacement, velocity, acceleration, tilting, rotation, and inversion.The plurality of buttons 2, 4, 6, 8, 10, plurality of vibration elements3, 5, 7, 9, 11, and the gyroscope 30 a are preferably associated with aprocessor 12. The plurality of buttons 2, 4, 6, 8, 10, the plurality ofvibration elements 3, 5, 7, 9, 11, and the gyroscope 30 a each provideinput signals to the processor 12, and the processor 12 sends outputsignals to each of the plurality of buttons 2, 4, 6, 8, 10, theplurality of vibration elements 3, 5, 7, 9, 11, and the gyroscope 30 a.

The processor 12 is preferably in electrical communication with theplurality of buttons 2, 4, 6, 8, 10, plurality of vibration elements 3,5, 7, 9, 11, and gyroscope 30 a such that data can be transmitted andreceived to and from the processor 12 from the components in the housing1. The processor 12 accepts data from the plurality of buttons 2, 4, 6,8, 10 for processing and transmission to a transmitter/receiver unit 13.The transmitter/receiver unit 13 transmits data to a central processingunit (CPU) or computer 40 for use in visualization software such asimmersive 3-D virtual reality software. For example, if the user pressesa combination of the buttons 2, 4, 6, 8, 10, then the processor 12transmits data corresponding to that respective combination of pressurefor the buttons 2, 4, 6, 8, 10 to the transmitter/receiver unit 13,which then transmits the data to the computer 40. The visualizationsoftware associated with the computer 40 then also generates tactilefeedback data, which can be transmitted by the computer 40 to thetransmitter/receiver unit 13. The transmitter/receiver unit 13 sendsthis data to the processor 12 which causes vibration elements 3, 5, 7, 911 to vibrate. The transmitter/receiver unit 13 serves as anuplink/downlink or transmitter/receiver between the processor 12 and thecomputer 40.

As shown in FIG. 3, a bearing 14 allows a user to freely rotate thehousing 1 through a range of motion when the user grips and operates thehousing 1. The bearing 14 could include any suitable joint-typearrangement, such as a ball and socket bearing, to allow the housing 1to move in multiple dimensions and through a range of motion. Forexample, the bearing 14 allows the housing 1 rotate, move forward,backward, left, right, tilt, and any combination thereof. As the userrotates the housing 1, then the gyroscope 30 a generates data related tothe three dimensional orientation of the housing 1 so that data can beused in visualization software in the computer 40 to rotate anassociated virtual hand in a virtual world. For example, a user can bendand twist the user's wrist while grasping the housing 1 and intuitivelycause similar movements of a virtual hand in a virtual world. Thehousing 1 preferably attaches to a grip 15 of a 3-D mouse 17 and afeedback controller 22 so that as the user moves the housing 1, then thegrip 15 of the 3-D mouse 17 moves accordingly and thetransmitter/receiver unit 13 sends data to visualization software. Thegrip 15 is designed to comfortably fit in the user's hand and caninclude buttons 15 a, 15 b, 15 c that are actuated to provide furtherfeedback to the user. In one embodiment, the grip 15 includes analoguebuttons. In one embodiment, the buttons are placed so that each of theuser's fingers rests comfortable on a corresponding button. The feedbackcontroller 22 holds and moves the grip 15 in accordance with physicalsensations that the user would feel and which are commensurate with theactions and experiences of the user's representation in the virtualworld. The grip 15 and feedback controller 22 each provide an additionalpoint of movement and/or pivoting for the housing 1, such that the grip15 and feedback controller 22 act as joints for the device 100, i.e.each of the components provide additional degrees of freedom for thedevice.

FIG. 1d shows a schematic diagram of the housing 1, the 3-D mouse 17,the grip 15, and the feedback control 22. As shown in FIG. 1d , the grip15 is connected to the housing 1 via a grip shaft 15′. In oneembodiment, the grip shaft 15′ is flexible. In one embodiment, the gripshaft 15′ is connected at a first end by a resilient ball-socket jointto the housing 1 and at a second end by a resilient ball-socket joint tothe grip 15. Based on this connection to the housing 1 and grip 15, thegrip shaft 15′ provides enough support to keep the housing 1 erect in aresting position, but provides additional degrees of motion for thedevice 100 when a user manipulates the housing 1. In one embodiment, thegrip shaft 15′ includes a grip sensor 15 a′ that detects deformationsand movement of the grip shaft 15′. The grip sensor 15 a′ collects datarelated to deformation and movement of the grip shaft 15′ and this datais used by the computer 40 to correspond the physical movement of thehousing 1 with the virtual reality element. The grip sensor 15 a′includes a plurality of sensors, including an accelerometer, gyroscopicsensor, torque sensor, strain gauge or any combination of known sensors.Similarly, feedback controller shafts 22 a, 22 b, 22 c, 22 d areprovided between the grip 15 and the feedback controller 22. Althoughfour feedback controller shafts 22 a, 22 b, 22 c, 22 d are shown in FIG.1d , one of ordinary skill in the art will recognize from the presentdisclosure that any number of shafts could be used. Similar to the gripshaft 15′, each of the feedback controller shafts 22 a, 22 b, 22 c, 22 dare preferably connected at each end to a respective component via aresilient ball-socket joint. This arrangement provides additionaldegrees of freedom for movement of the housing 1. The feedbackcontroller shafts 22 a, 22 b, 22 c, 22 d each include feedbackcontroller shaft sensors 22 a′, 22 b′, 22 c′, 22 d′, respectively. Thesefeedback controller shaft sensors 22 a′, 22 b′, 22 c′, 22 d′ eachincludes a plurality of types of sensors, including but not limited toan accelerometer, a gyroscopic sensor, a torque sensor, and a straingauge. One of ordinary skill in the art recognizes that other types ofsensors could be integrated into the feedback controller shafts 22 a, 22b, 22 c, 22 d to detect additional data points regarding the feedbackcontroller shafts 22 a, 22 b, 22 c, 22 d. The data detected by the gripsensor 15 a′ and the feedback controller shaft sensors 22 a′, 22 b′, 22c′, 22 d′ is sent to the computer 40 for plotting and mapping thedetected motions with respect to a virtual reality element. The datafrom the grip sensor 15 a′ and the feedback controller shaft sensors 22a′, 22 b′, 22 c′, 22 d′ can be processed by the device processor 12, andsent to the computer 40 via the transmitted/receiver unit 13.

In one embodiment, the visualization software includes immersive 3-Dvirtual reality software. In addition, if the grip 15 of the feedbackcontroller 22 receives forces related to feedback generated byvisualization software then those forces are imparted to the housing 1of the device and thereby felt by the hand of the user.

FIG. 4 one embodiment of the device as used as intended with thecomputer 40, the 3-D mouse 17, and the feedback controller 22. Thetransmitter/receiver unit 13 transmits data generated by the processor12. The transmitter/receiver unit 13 is preferably wireless, but a wiredor other communication could be used to transmit data and signalsbetween the components. A central processor 18 accepts data from thedevice processor 12 which is used to compute refined data related to thearticulation and orientation of the virtual hand, and the refined datais stored or used in software such as immersive 3-D virtual realitysoftware. The central processor 18 receives data from the device'stransmitter/receiver unit 13 though a second transmitter/receiver unit19 in the central processor 18. The central processor 18 connects to astorage device 20 and a display device 21, which provides visualfeedback to the user. The storage device 20 includes a memory unit, andthe memory unit stores multiple virtual reality scenarios, which can beloaded by the computer 40. The computer 40 is preferably connected tothe internet. The computer 40 can download or stream multiple virtualreality scenarios, which are displayed on the display device 21. Theuser manipulates the housing 1 to interact with a selected one of thevirtual reality scenarios. As used in this application, a virtualreality scenario can include, for example, an artificially createdlandscape that the user's point of view moves through as if the user ispresent in that particular landscape. In one embodiment, the userapproach a virtual reality tree in a virtual reality landscape, andmanipulate the tree via movement of the housing. In our system the userwould receive physical feedback through the device that indicate whenthe user's artificially created hand contacts the artificial tree andthat reflect the texture of the tree's surface.

FIG. 5 illustrates the steps for interaction between the housing 1 andthe computer 40. As shown in FIG. 5, step 500 includes a user grippingthe housing 1. Next, step 510 includes the user manipulating the housing1, which includes multiple degrees of motion, as well as displacement ofthe housing 1. Step 520 includes detecting the manipulation of thehousing 1, and converting data regarding the manipulation via theprocessor 12. Step 530 includes the first transmitter/receiver unit 13of the housing 1 transmitting data to the second transmitter/receiverunit 19 of the computer 40. Step 540 includes the central processor 18of the computer 40 analyzing the data transmitted from the housing 1 tomanipulate a virtual reality element such that the manipulation of thevirtual reality element corresponds with the manipulation of the housing1. Step 550 includes the display device 21 simultaneously displaying avirtual reality element in a virtual reality scenario as beingmanipulated in real-time based on the movement detected during step 510.Step 560 includes feedback data being sent by the secondtransmitter/receiver unit 19 to the first transmitter/receiver unit 13,and the housing 1 being moved and manipulated based on the feedbackdata. Step 570 includes the repetition of steps 510, 520, 530, 540, 550,and 560, in any order or combination of steps. One of ordinary skill inthe art recognizes that other steps may be included to provide forinteraction between the housing 1 and the computer 40. The steps areprovided in a continuous feedback loop, such that the user continuesmanipulating the housing 1 and the computer 40 continuously providesvisual feedback via the display device 21 and physical feedback viaimpulses sent to the housing 1 based on the user's manipulation of thehousing 1 and interaction of the virtual reality element in the virtualreality scenario.

Other embodiments will be obvious to one skilled in the art. Forexample, the buttons may be actuated as well as or instead of vibrating.Also, buttons may be replaced by touch sensors. Also, for example,direction changes can be detected by an accelerometer rather than agyroscope. I intend to include all such embodiments that are obvious toone with ordinary skill in the art.

What is claimed is:
 1. A system for dexterous interaction in a virtualworld, the system comprising: a manipulation device comprising: ahousing including a plurality of buttons and a plurality of vibrationelements each associated with at least one of the plurality of buttons;an orientation sensor configured for detecting data based on orientationof the housing; a processor in communication with the plurality ofbuttons, the plurality of vibration elements, and the orientationsensor, wherein the processor is configured to receive the data from theorientation sensor, and the processor is configured to transmit thedata; a computer comprising a display, wherein a virtual environmentincluding at least one virtual element adapted to be manipulated basedon movement of the manipulation device is shown on the display, and thecomputer receives the data from the processor and the computer isconfigured to analyze the data to manipulate the at least one virtualelement, and manipulation of a secondary virtual element by the at leastone virtual element in the virtual environment provides tactile feedbackto the manipulation device, and the tactile feedback is representativeof at least a surface characteristic of the secondary virtual element.2. The system of claim 1, wherein the housing is configured to move withthree degrees of freedom via a bearing.
 3. The system of claim 1,wherein the housing is coupled with a 3-D mouse and feedback controllervia a bearing.
 4. The system of claim 1, wherein the orientation sensorcomprises an accelerometer and a gyroscope.
 5. The system of claim 1,wherein the computer includes a storage device that includes a memoryunit, the memory unit includes a plurality of virtual environments, theplurality of virtual environments are displayed on the display, and auser interaction occurs with each one of the plurality of virtualreality scenarios based on movement of the manipulation device.
 6. Thesystem of claim 1, wherein the processor is configured to wirelesslytransmit and receive the data, and the computer is configured towirelessly transmit and receive the data.
 7. The system of claim 1,wherein the plurality of buttons are aligned with a user's fingers, andengagement of the plurality of buttons by the user results in acorresponding proportional manipulation of virtual reality fingers inthe virtual scenario displayed on the display.
 8. The system of claim 1,wherein the housing has an ellipsoid shape.
 9. The system of claim 1,wherein the housing is connected to a flexible shaft.
 10. The system ofclaim 1, wherein the housing is connected to a plurality of controllershafts each associated with a feedback controller shaft sensor.
 11. Thesystem of claim 1, wherein the plurality of vibration elements eachprovide sensory output associated with a virtual hand in visualizationsoftware.
 12. The system of claim 1, wherein the virtual element ismanipulated in real-time and in a continuous feedback loop based on thedata transmitted and received between the manipulation device and thecomputer.
 13. The system of claim 12, wherein the continuous feedbackloop provides physical feedback to the manipulation device based onmanipulation of the virtual element.
 14. The system of claim 1, whereinthe at least one virtual element is manipulated in real-time based onthe data corresponding to manipulation of the manipulation device.